Blended reality systems and methods

ABSTRACT

Systems and methods are provided for generating a blended reality view to a user, the blended reality view combining images reflected by a mirror with images transmitted from a screen behind the mirror. Systems for generating blended reality views can include a display device with a screen positioned behind a mirror. The display device can generate a pattern of illumination and non-illumination on the screen so that the illuminated portions of the screen substantially transmit through the mirror. Projectors can be used to illuminate objects in front of the mirror so that the illuminated objects are reflected by the mirror. In combination, the portions of the screen transmitted through the mirror and the illuminated objects reflected by the mirror can provide a blended reality view to a user viewing the mirror.

BACKGROUND

For entertainment and other purposes, unique visual displays can enhancethe experiences of users. These visual displays can be used to alterscenes as perceived by users, for example, by adding objects to thescene that do not actually exist. One method of providing such a visualdisplay uses an illusionary technique referred to as “Pepper's ghost”that can produce a virtual object in a scene as a latent or ghost-likeimage. This illusionary technique uses glass angled relative to aviewer, a display or object out of direct view of a viewer, and tailoredlighting schemes. However, such techniques are significantly limited,requiring space that extends beyond a visual display or thatsignificantly increase the size of a visual display. In addition, thesetechniques can produce unrealistic results as the virtual objects placedin the scene are generally translucent and low contrast.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and advantages of the embodiments provided herein are describedwith reference to the following detailed description in conjunction withthe accompanying drawings. Throughout the drawings, reference numbersmay be re-used to indicate correspondence between referenced elements.The drawings are provided to illustrate example embodiments describedherein and are not intended to limit the scope of the disclosure.

FIG. 1A illustrates an example blended reality apparatus configured togenerate a blended reality view.

FIGS. 1B and 1C illustrate a display of the example blended realityapparatus of FIG. 1A.

FIG. 2 illustrates a diagram of a user simultaneously perceivingreflected light and transmitted light, wherein the respectiveintensities of the reflected and transmitted light are controlled by ablended reality apparatus.

FIG. 3A illustrates a top view of an example blended reality apparatus,wherein the apparatus includes a plurality of projectors configured toselectively illuminate objects within an environment in front of themirror.

FIG. 3B illustrates a top view of an example blended reality apparatus,the apparatus including an active transmission matrix between a displayand a mirror.

FIG. 4 illustrates an example apparatus configured to generate a blendedreality view, the apparatus configured to control lighting within anenvironment to enhance the blended reality view.

FIG. 5A illustrates a top view of an example apparatus for generating ablended reality view, the apparatus including projectors and cameraspositioned around and within an environment in front of a mirror.

FIG. 5B illustrates a top view of a blended reality system comprising aplurality of blended reality apparatuses, each blended reality apparatusconfigured to generate a blended reality view.

FIGS. 6A and 6B illustrate functional block diagrams of example blendedreality systems comprising an image blending system.

FIG. 7 illustrates a functional block diagram of an example imagingblending system.

FIG. 8 illustrates an example method for generating a blended realityview.

DETAILED DESCRIPTION

Generally described, aspects of the present disclosure relate togenerating a blended reality view for a user by combining reflectionsfrom a mirror with light transmitted through the mirror by a display.The present disclosure includes systems and methods configured to blendtransmitted and reflected light to form a single scene, as perceived bya user, by controlling the amount of light on either side of a mirror orother reflective element. A blended reality view can be used to providea visual representation of the user in different settings other than theone the user is actually in. Similarly, the blended reality view can beused to provide a visual representation of items, such as clothes, onthe user without the user actually wearing the physical item.

Display systems can be used to provide a view of objects that are notactually in a scene but that are perceived to be there by a user. Thiscan be accomplished using angled glass and lighting techniques.Teleprompters, amusement park rides, heads up displays, visualillusions, and the like employ similar methods for providing a user aview of a portion of reality (e.g., light transmitted from a scenethrough the angled glass) along with a portion of projected objects(e.g., light reflected from an object or display by the angled glass).

However, challenges arise when it is desirable to combine a reflectionof a user with projected images of virtual objects. Using angled glasswith special lighting techniques does not provide such a view becausethe angled glass does not reflect an image of the user back to the user.In addition, attempting to combine a reflected view of the user with aview of projected images can result in undesirable superposition ofreflections with projected images, sometimes called “ghosting.” Withoutcontrolling which objects are seen as reflections by a user, thisundesirable combination of reflected images with transmitted orprojected images can result in an unclear image being perceived by theuser.

In addition, visual displays that employ angled glass to project imagesto a user can use additional space for a display placed out of thedirect line of site of the user. This can increase the size of a visualdisplay making it difficult or cumbersome to install and/or use.

Accordingly, the present disclosure provides systems and methods thatgenerate a blended reality view by controlling the amount of lighttransmitted through a mirror and the amount of light reflected from themirror. Blended reality apparatuses and methods described herein reduceor eliminate undesirable superposition of reflected and projected imagesby controlling the amount of light transmitted through a mirror and anamount of light reflected by the mirror. Using a blended realityapparatus as disclosed herein, a viewer can see reflected light in afirst region of a mirror, transmitted light in a second region of themirror, and a controlled superposition of reflected and transmittedlight in a third region of the mirror. Blended reality apparatusesdisclosed herein can be configured to control which portions of a mirrorprovide reflected light to a user, which portions of the mirror providetransmitted light to the user, and which portions of the mirror providea controlled superposition of reflected and transmitted light to theuser. This may be desirable in a situation where a user wants to see howa new outfit would look in a particular setting. To generate a blendedreality view, a blended reality apparatus can i) selectively illuminatethe user, ii) leave the rest of the room dark, and/or iii) project animage of the particular setting (e.g., a restaurant), the projectedimage being shown on the display at a location where the user would haveseen a reflection of objects or surfaces in the room. Because the useris illuminated, the user can see their reflection. Because the rest ofthe room is dark but the apparatus is projecting an image of theparticular setting, the user can see the setting instead of a reflectionof the room. Furthermore, the apparatus can also project an image of thenew outfit so that the user sees a superposition of their reflectionwith the projected new outfit so that it appears as though the user iswearing the outfit.

The blended reality systems and methods disclosed herein can provide ablended reality view using a mirror that is transmissive and reflective,a plurality of projectors to selectively illuminate objects in a room infront of the mirror, and a display to selectively transmit imagesthrough the mirror. To provide the blended reality view, projectors canselectively illuminate objects in a room and the display can be used toproject images through the mirror. Depending on the area of the mirrorthe user looks at, the light reaching the user's eye can be dominated bylight reflected by the mirror, dominated by light transmitted throughthe mirror, or a combination of reflected and transmitted light. Thus,the blended reality view can be a combination of reflected images andtransmitted images where the images are perceived by the user as asingle scene. Furthermore, the blended reality systems and methodsdisclosed herein can provide the blended reality view using an apparatusthat is compact relative to designs employing angled glass because thedisplay and mirror can be attached or otherwise combined with oneanother so that a surface of the mirror is adjacent to and parallel to asurface of the display.

Although the examples and implementations described herein focus, forthe purpose of illustration, on displays, projectors, cameras, andmirrors for generating blended reality views, one skilled in the artwill appreciate that the techniques described herein may be applied toother processes, methods, or systems. For example, the techniques may beused with other types of visual displays that collect and process imagedata for purposes other than providing a blended reality view to a user,but instead generate visual impressions for a user or a group of usersby combining transmitted images with reflections. Various aspects of thedisclosure will now be described with regard to certain examples andembodiments, which are intended to illustrate but not limit thedisclosure.

In one aspect, an apparatus can be configured to provide a user asimultaneous view of reflected and transmitted light to create a visualeffect for the user that blends reflections with displayed images. Theapparatus can include a mirror that is partially-reflective andpartially-transmissive and a display device positioned on a first sideof the mirror, the display device having a screen configured to generatelight transmitted through the mirror. The apparatus can include aplurality of projectors positioned on a periphery of the mirror andoriented to project light onto a user when the user is positioned on asecond side of the mirror, opposite the first side. The apparatus caninclude a camera configured to acquire image data of the user and theenvironment or scene on the second side the mirror. The apparatus caninclude an image blending system communicably coupled to the camera, thedisplay device, and the plurality of projectors. The image blendingsystem can be configured to determine a location of a user's eyes basedon the image data acquired with the camera, display images with thedisplay device so that portions of the screen are illuminated andportions of the screen are not illuminated, and control the projectorsto illuminate portions of the user and/or objects in the scene. When theuser views the mirror, the user sees a reflection from the mirror ofilluminated objects in the scene and the transmitted images from thedisplay device through the mirror, the transmitted images beingperceived as part of the reflected scene.

Overview of an Example Blended Reality Apparatus

Turning now to FIG. 1A, an example blended reality apparatus 100 isillustrated that is configured to generate a blended reality view 116for a user 105 by combining light reflected by a mirror 120 with lightfrom a display device 110 transmitted through the mirror 120. Theblended reality apparatus 100 includes a plurality of projectors 130around or near a frame 125 of the mirror. The plurality of projectors130 is configured to selectively illuminate objects in front of themirror 120. The blended reality apparatus 100 includes one or morecameras 140 on or near the frame 125 of the mirror. The one or morecameras 140 are configured to acquire image data of the user 105 andobjects in front of the mirror 120.

The blended reality apparatus 100 can generate the blended reality view116 by controlling the illumination of objects in front of the mirror120 using the projectors 130 and by generating images with the displaydevice 110 behind the mirror 120. The blended reality view 116 includesa combination of reflected light (e.g., a reflection 122 of the user)and transmitted light (e.g., a beach scene 112 or a shirt 117 projectedby the display device 110, as described herein with reference to FIGS.1B and 1C, respectively). To generate the blended reality view 116, theblended reality apparatus 100 can use the one or more cameras 140. In ascanning phase, the one or more cameras 140 can be used to acquire imagedata of the environment in front of the mirror 120, where theenvironment in front of the mirror 120 includes objects, surfaces,and/or light sources that can be seen reflected in the mirror 120. Insome embodiments, this first phase, or scanning phase, occurs during aninitial setup of the blended reality apparatus 100. From this imagedata, a three dimensional virtual model of the environment can begenerated. For example, simultaneous localization and mapping (“SLAM”)techniques can be used to generate the virtual model. SLAM techniquescan include acquiring image information of an environment to update anestimate of a position or positions of an object's location in theenvironment using Kalman filters, particle filters, and Monte Carlomethods. Other methods include combining information from opticalcameras, infrared cameras, and/or range finders to determine and tracklocations of objects. Examples of such techniques can be found, forexample, in U.S. Pat. No. 8,744,121 entitled “Device for Identifying andTracking Multiple Humans Over Time,” U.S. Pat. No. 8,717,417 entitled“Three-Dimensional Mapping and Imaging,” and U.S. Pat. Pub. No.2010/0302138 entitled “Methods and Systems for Defining and Modifying aVisual Representation,” each of which is incorporated herein byreference in its entirety. In some embodiments, the one or more cameras140 can be used to update the three dimensional virtual model from timeto time, such as when requested by a user, at regular intervals, and/orwhen changes to the environment are detected.

In addition, during a tracking phase, the one or more cameras 140 can beused to acquire image data of the user 105 when the user is using theblended reality apparatus 100. The image data of the user 105 can beused to track the user's face and/or to determine the location of theuser's eyes in relation to the mirror 120. The one or more cameras 140can each have a field of view. The one or more cameras 140 can beconfigured to image a user positioned within the field of view of atleast one of the one or more cameras 140. The field of views of one ormore cameras 140 can overlap to provide regions where image informationof the user is acquired from a plurality of angles. Accordingly, theblended reality apparatus 100 can perform the tracking phase when theuser 105 is positioned within the field of view of at least one camera.In some embodiments, tracking of the user's face and/or the user's eyesmay improve when the blended reality apparatus 100 includes a pluralityof cameras and the user is positioned within the field of view of atleast two cameras. In some embodiments, the blended reality apparatus100 is configured to track the user's face and/or to determine thelocation of the user's eyes when the user is positioned in front of themirror 120 such that the user 105 can see the user's own reflection(e.g., the user 105 is not positioned outside the frame 125 of themirror 120). Example systems and methods configured to track a humanhead, face, and/or form in real time are described, for example, in U.S.Pat. No. 8,594,425 entitled “Analysis of three-dimensional scenes,” andU.S. Pat. Pub. No. 2013/0202161 entitled “Enhanced Face Detection UsingDepth Information,” each of which is incorporated herein by reference inits entirety. In some embodiments, facial tracking is accomplishedthrough the use of the Kanade-Lucas-Tomasi (“KLT”) algorithm.

In some embodiments, the user's face can be tracked in real time or nearreal time with the one or more cameras 140. With the user's eyeslocated, reverse ray tracing (e.g., tracing rays of light from theuser's eye to the mirror and to the environment) or other techniques canbe performed to determine and associate the different parts of theenvironment the user can see. This information can then be used tocontrol the projectors 130 to selectively illuminate objects and/orsurfaces in the environment. This information can also be used tocontrol the display device 110 to selectively illuminate portions of themirror 120 from behind the mirror 120.

By way of analogy, in one embodiment, the mirror 120 can be thought ofas a single pixel-based screen showing two images to be blended. Thefirst image is the reflection of the user 122 and/or objects in theenvironment. Objects in the environment can include furniture, clothes,books, toys, devices, and the like. Objects in the environment can alsoinclude walls, floors, ceilings, or other structures. Objects in theenvironment can also include any surface, texture, or other featurevisible as a reflection in the mirror 120. The first image can becalculated based on the user's eye position from the image data acquiredwith the one or more cameras 140. For example, reverse ray tracing canbe done to determine the objects visible in the mirror using informationabout the user's eye position and the locations of objects in theenvironment. The second image is the virtual image 112 generated by thedisplay device 110 where the virtual image 112 is transmitted throughthe mirror 120. The blending of the two images can be accomplished bycontrolling the relative intensities of the light in both images. Forexample, light intensity and/or color at each pixel in the displayand/or light intensity and/or color projected by the projectors ondifferent objects can be varied to achieve a desired mixing of light atthe user's eye. At each “pixel” or location on the mirror 120, the imageseen by the user corresponds to the image providing the most light atthat pixel. For example, if the first image, or reflection, is to beseen at a particular location on the mirror 120 then the projectors 130can be configured to illuminate whatever object the user will see whenlooking at the mirror 120 at that particular location on the mirror 120.This determination is based on the location of the user's eyes and thethree dimensional virtual model of the environment. The display device110 can be dark, blank, display a solid color (e.g., black, white, blue,or other color), or display a selected pattern at the correspondinglocation. If on the other hand the second image, or virtual image, is tobe seen at a particular location on the mirror 120 then the displaydevice 110 can be configured to transmit light from behind the mirror120 using a pixel or collection of pixels that corresponds to thatparticular location on the mirror 120. The projectors 130 can beconfigured to not illuminate whatever object the user sees when lookingat the mirror 120 at that particular location on the mirror 120. As aresult, the user 105 sees at each “pixel” or location on the mirror 120reflected light or transmitted light. The user then perceives a blendingof the two images resulting in a single blended reality scene.

FIG. 2 illustrates a diagram of a user 105 simultaneously perceivingreflected light and transmitted light, wherein the respectiveintensities of the reflected and transmitted light are controlled by theprojectors 130, the display device 110, and/or ambient light sources 135a, 135 b. The blended reality apparatus 100 can be configured to controlthe light in the environment using the projectors 130. The light fromthe projectors 130 can be controlled to control reflected lightintensity at the mirror 120 on a “per pixel” basis meaning that theimage or light pattern projected by each projector 130 can be configuredto result in desired reflected light intensity at particular locationson the mirror 120. In some embodiments, the blended reality apparatus100 can also control ambient lighting in the environment using lights135 a, 135 b in the environment or lights outside of the environment.The lights 135 a, 135 b are generally not controllable on a “per pixel”basis because they are generally diffuse sources of light. However, theintensity and/or color of light from the lights 135 a, 135 b can becontrolled. Furthermore, the blended reality apparatus 100 can beconfigured to control transmitted light using the display device 110.The display device 110 can also be controlled on a “per pixel” basis.

As shown in FIG. 2, the user 105 will perceive an image based on thelight arriving at the user's eye. In the figure, solid lines 113, 114represent transmitted light from the display device 110 and dashed lines123, 124 represent light reflected from surfaces in the environment. Thethickness of the lines corresponds to the relative intensity of thelight. When looking at a particular region of the mirror 120, the userwill perceive a transmitted or virtual image if light from the displaydevice 110 transmitted through the mirror 120 in that region dominatesthe light from a surface 150 a reflected from that region of the mirror120 (e.g., transmitted light 113 dominates reflected light 123). Toincrease the contrast between transmitted light and reflected light, thesurface 150 a can be left unilluminated by the projectors 130. Thecontrast between transmitted light and reflected light can be thedifference in light intensity of the two sources. For example, thecontrast between transmitted light and reflected light can be expressedas (IT−IR)/(IT+IR), where IT is the intensity of transmitted light andIR is the intensity of reflected light. When looking at a particularregion of the mirror 120, the user will perceive a reflected image wherelight from a surface 150 b reflected at the particular location on themirror 120 dominates light transmitted by the display device 110 throughthe mirror at the particular location (e.g., the reflected light 124dominates transmitted light 114). To increase the contrast betweenreflected and transmitted light, the surface 150 b can be illuminated bythe projectors 130. In some situations, it may be undesirable for a userto see a combination of reflected light and transmitted light at asingle “pixel” or location on the mirror 120. This undesirablecombination can be reduced or eliminated by increasing the contrastbetween the reflected and transmitted light. To reduce or eliminateundesirable superposition of reflected and transmitted images, theintensity of the light that is not to be perceived can be reduced oreliminated. In the case of the display device 110, this can mean notilluminating that portion of the display device 110 or using an activeelement to block that portion of the display device 110. Examples ofsuch active elements are described in greater herein with reference toFIGS. 3B and 6B. In the case of reflections from objects in front of themirror, this can mean not illuminating the surface with the projectors130 and/or reducing or eliminating ambient lighting. In some situations,it may be desirable for a user to see a combination of reflected lightand transmitted light at a location on the mirror 120. The desirablecombination can be controlled by controlling the respective intensitiesof reflected and transmitted light. As used herein, intensity of lightcan be used to mean the number of photons per unit area per solid angle(e.g., radiant intensity), or power emitted by a light source weightedaccording to the sensitivity of the human eye (e.g., luminousintensity), brightness of a light source, or other such measurement usedto quantify the power emitted by a light source or perceived by aviewer. In some embodiments, the ratio of intensities of light can beconfigured to achieve a desired effect. For example, where the user seesreflected light, the ratio of intensities of reflected light totransmitted light can be about 2:1, about 3:1, about 4:1, about 5:1, orgreater than about 5:1. Similarly, where the user sees transmittedlight, the ratio of intensities of transmitted light to reflected lightcan be about 2:1, about 3:1, about 4:1, about 5:1, or greater than about5:1.

In some embodiments, it may be desirable to blend transmitted light withreflected light. This can be accomplished by varying the intensity ofillumination provided by the projectors 130 on a surface and theintensity of the light provided by the display device 110. The relativeintensities of reflected and transmitted light can be varied to producedesired effects, such as overlaying a virtual image on a reflectedimage. For example, this can be used to show the user 105 how an articleof clothing or make up would look on the user 105.

In some embodiments, the blended reality apparatus 100 is configured toilluminate the user 105 and objects in the environment so that the usersees the user's own reflection while reflections of objects in theenvironment are reduced or eliminated. This can be used to substantiallyisolate the user's reflection. This can reduce computational costs dueat least in part to the blended reality apparatus 100 not determiningwhat to display in each pixel or location of the mirror 120. Rather, theblended reality apparatus 100 determines the portions of the user toilluminate with the projectors 130 while leaving the rest of theenvironment without illumination from the projectors 130. In someimplementations, the display device 110 displays a substantially uniformcolor or pattern to help in isolating the reflection of the user 105.

To generate a blended reality view where the user sees the user's ownreflection along with an artificial scene, the display device 110 of theblended reality apparatus 100 can be configured to project an image orlight from targeted pixels and project no relatively little light atother targeted pixels. An example of this is illustrated in FIG. 1B.FIG. 1B illustrates the display device 110 of the blended realityapparatus 100 illustrated in FIG. 1A, with the mirror 120 and othercomponents removed. The display device 110 includes a screen 111 thatdisplays an image 112 (e.g., the beach scene 112) but with a portion ofthe screen 111 shown in black. This black portion 118 corresponds to theportion of the mirror 120 where a reflection of the user 105 is to beperceived by the user 105 rather than an image transmitted by thedisplay device 110. The black portion 118 can be configured to transmitlittle or no light so that the light reflected from the mirror 120 atthat location on the mirror 120 dominates the light transmitted throughthe mirror 120 at that location on the mirror 120. The size, shape,color, and other properties of the black portion 118 can changedepending on the targeted or desired blended reality view 116 providedto the user. For example, in some cases the black portion 118 caninclude projected images, colors, textures, patterns, or the like thatare intended to be mixed with reflected images so that the userperceives a combination of transmitted and reflected light. The shape ofthe black portion 118 can be determined based on the location of theuser's eyes as well as the desired or targeted blended reality view 116.For example, as described herein, the blended reality apparatus 100 canutilize reverse ray tracing or other techniques to determine whichportions of the display device 110 to illuminate and how to illuminatethose portions. The blended reality apparatus 100 can use images of theuser 105 acquired with the one or more cameras 130 to update the shapeof the user's body to change the shape of the black portion 118 as theuser 105 moves. The blended reality apparatus 100 can be configured toupdate the shape of the black portion 118 in real time.

To generate a blended reality view where the user sees an article ofclothing or other object superimposed on the user's body, the displaydevice 110 of the blended reality apparatus 100 can again be configuredto project an image or light from targeted pixels and project norelatively little light at other targeted pixels. An example of this isillustrated in FIG. 1C. The display device 110 in this scenario displaysan image of a shirt 117 such that the user 105 views the image of theshirt superimposed on the user's body. The image of the shirt 117 can beadjusted to fit the body of the user by adjusting the size of the shirtbased on an analysis of images of the body of the user acquired with theone or more cameras 140. In some embodiments, a visual approximation canbe made to adjust the properties of the shirt 117 as displayed by thedisplay device 110. The visual approximation can be based on images ofthe shirt acquired when the shirt was worn by another person, whereinthe images of the shirt are adjusted based on the body of the user 105.The concepts described with reference to FIGS. 1B and 1C can becombined. For example, the blended reality apparatus 100 can beconfigured to provide a blended reality view that includes a projectedscene to change an environment around the user 105 (e.g., the beachscene 112 illustrated in FIG. 1B) and an object to be perceived as beingworn by the user 105 (e.g., the shirt 117 illustrated in FIG. 1C).

Additionally, the blended reality apparatus 100 can be configured tosubstantially isolate a reflection of the user 105 such that areflection of environment around the user is suppressed. For example,this may be accomplished by using the display device 110 to project atailored scene 112 around the user 105 (e.g., a substantially uniformlylit scene, a scene that is substantially white, a scene that issubstantially black, a scene lacking sharp details, a scene lackingdiscernible objects, or the like) and/or by not illuminating objects inthe environment with the projectors 130. By isolating the user'sreflection, the user 105 may be able to focus more on the user's ownreflection due at least in part to a reduction of potentiallydistracting objects being viewed by the user 105 in the mirror 120. Thiscan also reduce the computational costs associated with generating ablended reality view that includes objects or scenes to be viewed alongwith the reflection of the user 105.

Returning to FIG. 1A, the display device 110 of the blended realityapparatus 100 is positioned on a first side of the mirror 120. In use,the user 105 can be positioned on a second side of the mirror 120,opposite the first side, so that the user 105 can see the user's ownreflection in the mirror 120. The display device 110 can be a devicehaving a screen, such as an LCD or plasma television, or the displaydevice 110 can be a projector. In some embodiments, the display device110 is configured to be parallel to the mirror 120. In some embodiments,a screen of the display device 110 is configured to have a width that issubstantially the same width as the mirror 120 and a height that issubstantially the same height as the mirror 120. In some embodiments,the display device 110 can be configured to project images that cansubstantially cover the surface area of the mirror 120. For example andwithout limitation, the width of the screen or the width of theprojected image can be at least 80% and/or less than or equal to about120% the width of the mirror 120, at least 90% and/or less than or equalto about 110% the width of the mirror 120, at least 95% and/or less thanor equal to about 105% the width of the mirror 120, or at least 97%and/or less than or equal to about 103% the width of the mirror 120.Similarly, the height of the screen or the height of the projected imagecan be at least 80% and/or less than or equal to about 120% the heightof the mirror 120, at least 90% and/or less than or equal to about 110%the height of the mirror 120, at least 95% and/or less than or equal toabout 105% the height of the mirror 120, or at least 97% and/or lessthan or equal to about 103% the height of the mirror 120. In someembodiments, the display device 110 can be made up of multiple displaysand/or screens.

The display device 110 can be attached to the mirror 120 to form aunitary structure. In some embodiments, the display device 110 can beattached to the mirror 120 so that the display device 110 can be removedor replaced without damaging the mirror 120. The display device 110 canhave a surface that is adjacent to and parallel to a surface of themirror 120. In this configuration, the blended reality apparatus 100 canhave a depth that is a combination of the thickness of the mirror 120and the thickness of the display device 110. This can result in arelatively compact apparatus suitable for installation in manyenvironments, such as in a fitting room, in a closet, and/or hung on awall.

The mirror 120 can be a sheet of material that is reflective andtransmissive, such as glass or acrylic. The mirror 120 can be treated tobalance transmittance, reflectance, and absorptance properties toachieve targeted or desired results. The mirror 120 can be ahalf-silvered mirror, similar to a beam splitter, that has balancedreflectance and transmittance properties in the visible spectrum (e.g.,the mirror's reflectance and transmittance are about equal in thewavelength range between 350 nm and 700 nm). The mirror 120 can beplanar or it can have a curved surface. The front side of the mirror 120can be designated as the side of the mirror 120 facing the user 105 whenthe user 105 is viewing their reflection. With this convention, thedisplay device 110 is positioned facing the back side of the mirror 120so that light emitted from the display device 110 passes through themirror 120 from the back side to the front side to eventually reach theuser 105. The mirror 120 can include the frame 125 to provide supportfor the mirror 120 as well as providing a place to attach the one ormore cameras 140 and/or the projectors 130.

The projectors 130 of the blended reality apparatus 100 can bepositioned around the mirror 120, such as integrated into the frame 125of the mirror 120. The projectors 130 can be configured to project lightinto the environment in front of the mirror (e.g., where the user 105stands when perceiving the blended reality view 116). As described ingreater detail herein, the projectors 130 can be positioned at otherlocations. The blended reality apparatus 100 can include at least 2projectors, at least 3 projectors, at least 4 projectors, at least 5projectors, at least 6 projectors, at least 7 projectors, at least 8projectors, more than 8 projectors, or less than 10 projectors. In someembodiments, the projectors 130 can be positioned along at least twoorthogonal axes. For example, projectors can be positioned at themidpoints of the frame 125 or at the corners of the frame 125. Theprojectors 130 can be configured to generate a combined light output asif positioned at the user's eyes to achieve targeted or desired lightingeffects. The combined light output from the projectors 130 can beconfigured to illuminate desired or targeted surfaces in the environment(including the user) and to leave other surfaces without illuminationfrom the projectors. In some embodiments, the projectors 130 can becoupled to motors or other actuators to change an orientation of theprojectors 130. This can be done to change the pointing direction of theprojectors 130 to enable illumination of different objects.

The one or more cameras 140 of the blended reality apparatus 100 can beattached to the mirror 120 or positioned elsewhere. The one or morecameras 140 can be configured to acquire image data of the environmentin front of the mirror 120. The one or more cameras 140 can beconfigured to acquire image data of the user 105 when positioned in theenvironment in front of the mirror 120. The one or more cameras 140 caninclude image acquisition devices sensitive to different portions of theelectromagnetic spectrum. For example, at least one camera can besensitive to light in the visible portion of the spectrum and at leastone camera can be sensitive to light in the infrared portion of thespectrum. The one or more cameras 140 can also include time of flightcameras or other similar sensors configured to determine distances toobjects or surfaces in the environment. The one or more cameras 140 caninclude depth finding or range finding devices configured to determine adistance to one or more objects within a field of view of the respectivedevice. In some embodiments, the one or more cameras 140 can be coupledto motors or other actuators to change an orientation one or more of thecameras 140. This can be done to change the pointing direction of one ormore of the cameras 140 to acquire images of different objects and/or totrack movements of the user or objects.

The information acquired with the one or more cameras 140, such as imageand/or depth information, can be used to construct a virtual model ofthe environment for the blended reality apparatus 100. The virtual modelcan include a digital representation of objects and surfaces in theenvironment and their relative positions, orientations, colors,reflectivities, and the like. The virtual model can be used to determinethe patterns of light or images to be generated by the one or moreprojectors 130 to achieve a targeted or desired reflected image for theuser 105. Example systems and methods for constructing three dimensionalvirtual models of an environment using image data are described in U.S.Pat. No. 8,594,425 entitled “Analysis of three-dimensional scenes,” U.S.Pat. No. 8,717,417 entitled “Three-Dimensional Mapping and Imaging,”U.S. Pat. No. 8,326,025 entitled “Method for Determining a Depth Mapfrom Images, Device for Determining a Depth Map,” and U.S. Pat. No.8,649,025 entitled “Methods and Apparatus for Real-Time Digitization ofThree-Dimensional Scenes,” each of which is incorporated herein byreference in its entirety.

In some embodiments, the blended reality apparatus 100 can be configuredto perform a light-based scan of the environment to determine propertiesof the environment. This information can be used to determine the lightto be projected by the projectors 130 and/or ambient lights. In certainimplementations, the blended reality apparatus 100 performs this scaninstead of constructing a virtual model of the environment. This canreduce the computational costs of generating a blended reality view. Asan example, the blended reality apparatus 100 can be configured toincrementally increase or decrease the intensity of ambient lights orprojectors 130 to illuminate the environment. The one or more cameras140 can acquire images of the environment at the different levels ofillumination. Based at least in part on the acquired images, the blendedreality apparatus 100 can be configured to determine properties of theenvironment (e.g., color, reflectivity, etc.). Using these properties,the blended reality apparatus 100 can determine the targetedillumination to be provided by the projectors 130 to generate a targetedblended reality view. In certain implementations, the blended realityapparatus 100 controls the projectors 130 and/or ambient lights toilluminate targeted portions of the environment to characterize theoptical properties of those targeted portions. Again, using thecharacterized optical properties of the environment (e.g., color,reflectivity, etc.), the blended reality apparatus 100 can determine thetargeted illumination to be provided by the projectors 130 to generate atargeted blended reality view. In some implementations, the blendedreality apparatus 100 can be configured to use depth informationacquired with the one or more cameras 140 to determine the illuminationpattern provided by the projectors 130. For example, the depthinformation can be used to determine which objects are backgroundobjects and which objects are foreground objects (e.g., such as the user105). The blended reality apparatus 100 can then illuminate theforeground objects with the projectors 130 while leaving the backgroundobjects unilluminated. In some implementations, the blended realityapparatus 100 can be configured to project patterns of light ontosurfaces in the environment to generate depth maps of the environment.Examples of systems and methods for generating depth maps usingprojected light and/or acquired images of an environment are describedin U.S. Pat. No. 8,493,496 entitled “Depth Mapping Using ProjectedPatterns,” U.S. Pat. No. 8,326,025 entitled “Method for Determining aDepth Map from Images, Device for Determining a Depth Map,” and U.S.Pat. No. 8,649,025 entitled “Methods and Apparatus for Real-TimeDigitization of Three-Dimensional Scenes,” each of which is incorporatedby reference herein in its entirety.

The one or more cameras 140 can also be used to identify and track aface of the user 105 to determine the location of the user's eyes. Anysuitable method or algorithm can be used to accomplish this. Forexample, an infrared beam and an infrared camera can be used todetermine the location of the users' eyes based on reflected infraredlight from the retinas. Other facial and/or eye tracking systems andmethods are described in U.S. Pat. No. 8,408,706 entitled “3D GazeTracker,” U.S. Pat. No. 6,578,962 entitled “Calibration-Free Eye GazeTracking,” and U.S. Pat. No. 7,197,165 entitled “Eye Tracking UsingImage Data,” each of which is incorporated herein by reference in itsentirety.

The blended reality apparatus 100 can be used in a number of ways. Afirst example involves reflecting the head and shirt of a user andtransmitting an image of clothing on the rest of the user, therebyproviding a blended reality view that shows the user what the user waswearing the last time the user wore that shirt. To accomplish this, theblended reality apparatus 100 can control the projectors 130 toilluminate the head and shirt of the user so that they are reflected tothe user and control the display device 110 to transmit an image orimages of the other articles of clothing the user was wearing. A secondexample involves providing a reflection of the user along withtransmitted scenes of different environments. This may be desirable whenthe user is trying on clothing to provide the user a view of how theoutfit would look in different settings. The blended reality apparatus100 can control the projectors 130 to illuminate the user so that theuser sees their own reflection and control the display device 100 totransmit images of different settings. The result can be that the actualsurroundings of the user (e.g., the user's closet or a dressing room)are replaced with images of the beach, a restaurant, an office, or thelike. The ambient lights can also be adjusted to accomplish a targetedor desired effect.

FIG. 3A illustrates a top view of the example apparatus 100 of FIG. 1A,wherein the apparatus 100 includes a plurality of projectors 130configured to selectively illuminate surfaces 150 a-d within anenvironment in front of the mirror 120. The projectors 130 can behigh-contrast projectors configured to project a range of visiblewavelengths with high dynamic range. The projectors include modulatingpanels (e.g., liquid crystal on silicon panels, liquid crystal devicepanels, digital micromirror device panel, etc.) configured to generatepatterns of light for projection. The projectors 130 can be configuredto project light at targeted areas. As an example, to illuminate object150 c so that its reflection is seen in the mirror 120 by the user 105,each projector 130 can determine (e.g., through an analysis of thevirtual model of the environment created by the blended realityapparatus) the pattern of light necessary to illuminate object 150 c butto not illuminate, for example, object 150 d. The combined light outputfrom the projectors 130 can thus illuminate targeted objects or surfacesin the environment while leaving others dark.

FIG. 3B illustrates a top view of the example apparatus 100, theapparatus 100 including an active transmission matrix 115 between thedisplay 110 and the mirror 120. The active transmission matrix 115comprises an active electronic element that can change the opticalproperties of individual pixels in the matrix. For example, the activetransmission matrix 115 can be a liquid crystal matrix configured tocontrol individual pixels, changing selected pixels from opaque totransparent (e.g., a pixel will block or transmit light) to furtherregulate the light transmitted from the display device 110 through themirror 120. This can improve control over reflected and transmittedlight perceived by the user. The active transmission matrix 115 can alsobe a material that can selectively change individual pixels from opaqueto reflective or from transmissive to reflective. In some embodiments,the active transmission matrix 115 can be integrated with the mirror 120or integrated with the display device 110. In some embodiments, theactive transmission matrix 115 is the mirror 120 such that the mirror120 can change the reflectance of individual pixels on a face of themirror 120. For example, the active transmission matrix 115 can includea liquid crystal switchable mirror comprising a solid state thin filmdevice that can be configured to switch between reflective, partiallyreflective, and transparent states. Examples of such apparatuses aredisclosed in U.S. Pat. No. 6,999,649 entitled “Optical Switches Made byNematic Liquid Crystal Switchable Mirrors, and Apparatus ofManufacture,” which is incorporated herein by reference in its entirety.This implementation advantageously provides greater control overreflected and transmitted images seen by the user. The activetransmission matrix 115 and the projectors 130 can thus combine toenhance the blended reality image by providing greater control over thetransmission and reflection of light at the mirror 120.

FIG. 4 illustrates an example apparatus 400 configured to generate ablended reality view, the apparatus 400 configured to control lightingwithin an environment to enhance the blended reality view. The blendedreality apparatus 400 includes hue lights 133 around a periphery of themirror. The hue lights 133 can be controlled to change the color and/orintensity of light within a room and/or on a wall behind the blendedreality apparatus 400 to enhance the generated blended reality view. Thehue lights 133 can be any suitable light or combination of lightsproviding a single color or a range of colors in the visible spectrum.For example, the color and/or intensity of the hue lights 133 can beconfigure to change based on the blended reality view perceived by theuser at the mirror 120. If the blended reality view includes the user ona boat in the ocean, the hue lights 133 can be configured to shine blueshimmering light on the walls, floor, and/or ceiling or other objects toenhance the blended reality experience. The hue lights 133 can includestrands of LED lights around a frame of the blended reality apparatus100 that change depending on what is transmitted and/or reflected by theapparatus 100.

The blended reality apparatus 400 can also be configured to controllighting fixtures 135. The lighting fixtures can be sources of diffuselight or directed light positioned throughout the environment in whichthe blended reality apparatus 400 will be used. The lighting fixtures135 can provide isotropic, diffuse, anisotropic, and/or directionallight (e.g., spotlights). However, the lighting fixtures 135 and the huelights 133 differ from the projectors 130 in that the projectors 130 canmodulate the light they produce to provide a targeted or desired lightoutput (e.g., by modulating pixels to project a pattern of light thatcan change over time) whereas the lighting fixtures 135 and/or huelights 133 can be controlled to change a direction, brightness and/orcolor of the light, but not to generate a targeted pattern of lightoutput.

FIG. 5A illustrates a top view of an example blended reality apparatus500 for generating a blended reality view, the blended reality apparatus500 including projectors 130 and cameras 140 positioned around andwithin an environment in front of a mirror 120. The projectors andcameras 140 positioned in such a manner can improve the generatedblended reality view. For example, cameras 140 positioned around theenvironment can be used to improve the scan of the environment and theresulting virtual model. An improved virtual model can be used toimprove control of the projectors 130 to more exactly illuminate desiredsurfaces. Similarly, projectors 130 positioned around the environmentcan be used to improve the targeted illumination of surfaces byproviding additional angles and optical pathways to surfaces that mayotherwise be occluded. The cameras 140 and projectors 130 can bepositioned at various heights with various pointing angles. For example,cameras 140 and/or projectors 130 can be positioned on ceilings, walls,objects, floors, and/or other surfaces. This can advantageously increasethe ability to selectively illuminate objects 150 a-d and/or notilluminate these objects.

In some embodiments, the additional cameras 140 and projectors 130 canbe used to provide additional views of the user in addition to and/orinstead of reflected views of the user. For example, cameras 140positioned behind the user can provide a rear view of the user so thatthe user can see how clothes appear from that angle.

FIG. 5B illustrates a top view of a blended reality system 550comprising a plurality of apparatuses 500 a-d, each blended realityapparatus 500 a-d configured to generate a blended reality view for auser 105. Each blended reality apparatus 500 a-d can include a displaydevice 110, a mirror 120, projectors 130, and one or more cameras 140.Each blended reality apparatus 500 a-d can operate independently togenerate blended reality views for the user. In some embodiments, theblended reality apparatuses 500 a-d are communicably coupled so thateach apparatus 500 a-d can receive, for example, image data from thecameras 140 of the other apparatuses and/or control the projectors 130of the other apparatuses. For an individual blended reality apparatus,this can provide the advantages described with respect to the blendedreality apparatus 500 described herein with reference to FIG. 5A. As anexample use, the blended reality system 550 can be used in retailoutlets in dressing rooms or other areas where users try on clothesprior to purchasing them. The blended reality system 550 can be used toprovide a plurality of simultaneous blended reality views for the user,generating an immersive blended reality environment.

FIG. 6A illustrates a functional block diagram of an example blendedreality apparatus 600 a comprising an image blending system 660. Theblended reality apparatus 600 a is configured to acquire image data withthe camera(s) 140, process that image data with the image blendingsystem 660, and to control the light output of the projectors 140 and/orother lighting (e.g., lighting fixtures or hue lights) to control whichobjects in an environment are reflected to a user for viewing. Inaddition, the image blending system 660 controls the display device 110to generate images at a screen 111 for transmission through the mirrorto the user. The combination of the reflected light and the transmittedlight forming a blended reality view based on the image informationprovided by the camera(s) 140. The image information provided by thecamera(s) 140 to the image blending system 660 can include scans of theenvironment in front of the mirror 120, images of the user, and/or depthinformation for objects and/or surfaces in the environment. In someembodiments, the camera(s) 140 can detect motion of the user or otherobjects in the environment to allow the image blending system 660 toactively compensate for such movement in real time.

FIG. 6B illustrates a functional block diagram of an example blendedreality apparatus 600 b comprising the image blending system 660 with anadditional active transmission matrix 115 relative to the blendedreality apparatus 600 a described with reference to FIG. 6A. The activetransmission matrix 115 can further be controlled by the image blendingsystem 660 to provide greater control over the transmission of lightfrom the screen 111 of the display device 110 and/or to provide greatercontrol over the reflective properties of the mirror 120.

FIG. 7 illustrates a functional block diagram of an example imagingblending system 660. The image blending system 660 can receive inputfrom the camera(s) 140 as well as user input 695. The user input 695 canbe received via voice commands, motion-based commands, touch interfacecontrols, wireless signals, or the like. The image blending system 660can include a controller 665 and data storage 670 configured torespectively execute and to store computer executable instructions forperforming the functions described herein. The components and modules ofthe image blending system 660 communicate with one another viacommunication bus 692 that can include wired communication, wirelesscommunication, or a combination of wired and wireless communication. Insome embodiments, the components and modules of the image blendingsystem 660 form a single computational system comprising computationalhardware. In some embodiments, components of the image blending system660 can be distributed among other components of a blended realityapparatus. For example, eye tracking functionality can be implemented byan eye tracking module 675 that can be include in one or more cameras140.

The image blending system 660 includes an eye tracking module 675configured to analyze image data acquired by the cameras 140 and todetermine an eye location of a user. The eye tracking module 675 can useany suitable method for determining eye position such as using an IRbeam and IR camera to identify retinal reflections. Other examplemethods and systems are described in U.S. Pat. No. 8,408,706 entitled“3D Gaze Tracker,” U.S. Pat. No. 6,578,962 entitled “Calibration-FreeEye Gaze Tracking,” and U.S. Pat. No. 7,197,165 entitled “Eye TrackingUsing Image Data,” each of which is incorporated herein by reference inits entirety. In some embodiments, the eye tracking module 675 isconfigured to track eye movements of the user in real time or in nearreal time. For example, the eye tracking module 675 can be configured totrack eye movement of the user with sufficient speed to allow a blendedreality apparatus to modify the transmitted and/or reflected light atthe mirror so that the user does not notice any lag between the user'smovement and updates to the blended reality view provided.

The image blending system 660 includes an environment scan module 680configured to analyze image data acquired with the cameras 140 of theenvironment around a blended reality apparatus. The environment scanmodule 680 can receive image data from the cameras of one or more viewsof the environment where the blended reality apparatus is used todetermine positions and orientations of objects and/or surfaces in theenvironment. The environment scan module 680 can then construct adigital representation of the environment using a three dimensionalvirtual model. This model can be used to perform reverse ray tracing orother techniques by the image blending system to determine what light totransmit by the display 110 and what patterns to project with theprojectors 130. The environment scan module 680 can be configured toscan the environment upon an initial setup, when commanded by a user,when the image blending system 660 identifies changes to theenvironment, or at designated times or intervals. The environment scanmodule 680 can be configured to compare current image data to previouslyacquired image data to determine if there have been changes to theenvironment. If changes are detected, the environment scan module 680can generate a new three dimensional virtual model of the environment.In this way, the virtual model can be updated to provide updated blendedreality views when objects or surfaces change in the environment.

The image blending system 660 can include a display module 685configured to determine the pattern of light to transmit by the display110 and/or the pattern of pixels to turn off or on by the activetransmission matrix. The display module 685 can analyze the virtualmodel along with the image data from the camera(s) 140 to determinewhere the user's eyes are and what virtual images are to be provided forthe targeted or desired blended reality view. The display module 685 cancommunicate with the display(s)/active transmission matrix 110 tocontrol transmission of light based on this information.

The display module 685 can be configured to provide different views fora left and a right eye of a user. This can be used to providestereoscopic views for the user enhancing the sensation of depth in thetransmitted images. In some embodiments, the display 110 can include alenticular lens array and/or use polarized light in conjunction withpolarized glasses to provide the stereoscopic experience for the user.In some implementations, the display module 685 can be configured toselectively blur images based on where the user is looking. This can bedone to enhance the effect of depth in a transmitted scene. In certainimplementations, blurring can be used by the display module 685 tosimulate the depth of field of the user's vision to compensate for focaldistances to images that the user is not focusing on.

The image blending system 660 can include a lighting module 690configured to control each projector and/or other lighting element 130to generate the desired or targeted light patterns. The targeted lightpatterns can be configured to selectively illuminate the user and/orother objects or surfaces in the environment so that the user sees thereflection of those objects in the mirror. For each projector 130, thelighting module 690 can determine the targeted light pattern byanalyzing the virtual model generated by the environment scan module680. For example, knowing the position of a particular projector, thelighting module 690 can be configured to determine a light pattern thatshines light on a targeted surface while leaving other surfaces withoutillumination. The lighting module 690 can determine the light output foreach projector 140 as well as the combined light output of a pluralityor all the projectors 140 to determine the final lighting effect. Thelighting module 690 can thus adjust the light output (e.g., lightpattern, light intensity, and/or light color) to selectively illuminateobjects so they are viewed as reflections in the mirror by the user.

In some embodiments, the lighting module 690 and the display module 685can work together to supplement and/or enhance the output of theprojectors 130 and the display 110. For example, where the displaymodule 685 controls the display 110 to transmit images of a beach scene,the lighting module 690 can generate complementary light patterns toimprove the visual appearance of the beach scene by projecting the lightpatterns on a wall or other surface. The result being that the user seesa combined transmitted image and a reflected image of a beach scene,enhancing the immersive and realistic quality of the blended realityview. Similarly, the display module 685 can control the display 110 totransmit images of objects in the room with modifications while thelighting module 690 can control the projectors to selectively illuminatethe objects. The result being that the user sees the reflection of theobjects combined with transmitted images on those objects, thusenhancing the blended reality view.

FIG. 8 illustrates an example method 800 for generating a blendedreality view. The method can be performed by any of the blended realityapparatuses or systems described herein. For ease of description, themethod will be described as being performed by a blended realityapparatus, but each step or combination of steps in the method can beperformed by a single component or a combination of components in theblended reality apparatus or system.

In block 805, the blended reality apparatus scans an environment withone or more cameras. The environment can include a room or otherlocation in which the blended reality apparatus is positioned. In someembodiments, the environment includes all objects and/or surfaces infront of the mirror, where the front of the mirror is the part of themirror viewed by a user to view their reflection. In some embodiments,the environment includes additional areas that are not in front of themirror or which are not visible as reflections when positioned in frontof the mirror. The scan of the environment can include acquiring imagedata, depth information, and/or other information about the positions,orientations, and/or optical characteristics of objects and surfaceswithin the environment.

In block 810, the blended reality apparatus generates a virtual model ofthe scanned environment. The virtual model can be a digitalrepresentation of the objects and surfaces within the environment, wherethe positions and orientations of the objects and surfaces arerepresented. The virtual model can include information about the opticalcharacteristics of those objects and surfaces for purposes of reverseray tracing when generating a blended reality view. In someimplementations, the virtual model includes information about opticalcharacteristics and/or positions of surfaces of the scanned environment,but it is not a digital representation of the objects and surfaceswithin the environment.

In block 815, the blended reality apparatus uses image data from one ormore cameras to identify a face and/or eyes of a user. The blendedreality apparatus can track the user's face and/or eyes in real time toupdate the blended reality view to compensate for movements of the user.

In block 820, the blended reality apparatus determines objects orsurfaces to be seen as reflections by the user. To generate the blendedreality view, the apparatus blends reflections of objects and/orsurfaces in the environment, including portions of the user, withtransmitted images. In block 825, the blended reality apparatusgenerates images to be transmitted by a display through a mirror to theuser. In block 830, the blended reality apparatus projects light ontothe user and/or surfaces in the environment so that these things will beviewed as reflections in the mirror. By controlling the amount, quality,and pattern of light projected by each projector, the reflected imagescan be controlled selectively reflect portions of the user and/orenvironment. The reflected images and transmitted images can beperceived by the user simultaneously, resulting in a blended realityview. After performing the steps in blocks 825 and 830, the methodreturns to block 815 to update the information related to the positionof the face and/or eyes of the user. The method can thus track in realtime the position of the face and/or eyes of the user to update theblended reality view so that the view changes as the user moves.

Depending on the embodiment, certain acts, events, or functions of anyof the algorithms described herein can be performed in a differentsequence, can be added, merged, or left out altogether (e.g., not alldescribed acts or events are necessary for the practice of thealgorithm). Moreover, in certain embodiments, acts or events can beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, and algorithm stepsdescribed in connection with the embodiments disclosed herein can beimplemented as electronic hardware, computer software, or combinationsof both. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. The described functionality can be implemented invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the disclosure.

The various illustrative logical blocks and modules described inconnection with the embodiments disclosed herein can be implemented orperformed by a machine, such as a processor configured with specificinstructions, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A processor can be amicroprocessor, but in the alternative, the processor can be acontroller, microcontroller, or state machine, combinations of the same,or the like. A processor can also be implemented as a combination ofcomputing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration.

The elements of a method, process, or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of computer-readablestorage medium known in the art. An exemplary storage medium can becoupled to the processor such that the processor can read informationfrom, and write information to, the storage medium. In the alternative,the storage medium can be integral to the processor. The processor andthe storage medium can reside in an ASIC. The ASIC can reside in a userterminal. In the alternative, the processor and the storage medium canreside as discrete components in a user terminal. A software module cancomprise computer-executable instructions which cause a hardwareprocessor to execute the computer-executable instructions. Thecomputer-executable instructions can comprise a scripted computerlanguage and/or a compiled computer language. Computer-executableinstructions can comprise, for example and without limitation,JAVASCRIPT®, PYTHON™, php, SQL, C, C++, JAVA®, C#, Fortran, BASIC, shellscripts, Perl, or the like.

Conditional language used herein, such as, among others, “can,” “might,”“may,” “e.g.,” and the like, unless specifically stated otherwise, orotherwise understood within the context as used, is generally intendedto convey that certain embodiments include, while other embodiments donot include, certain features, elements and/or states. Thus, suchconditional language is not generally intended to imply that features,elements and/or states are in any way required for one or moreembodiments or that one or more embodiments necessarily include logicfor deciding, with or without author input or prompting, whether thesefeatures, elements and/or states are included or are to be performed inany particular embodiment. The terms “comprising,” “including,”“having,” “involving,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Disjunctive language such as the phrase “at least one of X, Y or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y or Z, or any combination thereof (e.g., X, Y and/or Z).Thus, such disjunctive language is not generally intended to, and shouldnot, imply that certain embodiments require at least one of X, at leastone of Y or at least one of Z to each be present.

Unless otherwise explicitly stated, articles such as “a” or “an” shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it will beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As will berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. An apparatus to provide a view of reflected andtransmitted light to create a visual effect that combines reflectedimages and transmitted images, the apparatus comprising: a displayconfigured to display images; a mirror having a first side and a secondside, the mirror being partially-reflective and partially-transmissivesuch that at least a portion of light provided by the display is visiblefrom the first side of the mirror; a frame surrounding the mirror; aplurality of projectors positioned on the frame and configured toilluminate the user and object elements facing the first side of themirror; a camera configured to acquire image data of the user and objectelements; and an image blending system communicably coupled to thedisplay, to the plurality of projectors, and to the camera, the imageblending system configured to: in a scanning phase, analyze image datareceived from the camera to determine locations of the user and objectelements; in a tracking phase, analyze the image data received from thecamera of the user to determine a location of the user's eyes, thelocation of the user's eyes including a height and a distance of theuser's eyes relative to the mirror; control the plurality of projectorsto illuminate at least some of the user and object elements to bereflected by the mirror as viewed from a particular projector; andcontrol the display to display images that are visible from the firstside of the mirror within a first portion of the mirror; wherein lightfrom the illuminated user and object elements is reflected by the mirrorand is visible from the first side of the mirror within a second portionof the mirror and combine with the displayed images to form a compositeimage.
 2. The apparatus of claim 1 further comprising a lightingfixture, wherein the image blending system is further configured tocontrol the intensity of light output by the lighting fixture to enhancethe composite image formed from a combination of the reflected imagesand the displayed images.
 3. The apparatus of claim 1 further comprisingan active transmission matrix comprising a two-dimensional array ofpixels, the active transmission matrix positioned between a screen ofthe display and the mirror, wherein the image blending system is furtherconfigured to control the transmissive properties of the two-dimensionalarray of pixels of the active transmission matrix to control lighttransmitted through the mirror to improve control of a contrast betweenthe reflected images and the displayed images.
 4. A method of generatinga combined image at a targeted location that combines reflected imagesand transmitted images, the method comprising: acquiring image data ofsurfaces in an environment; generating a virtual three dimensional modelof the surfaces in the environment by analyzing the acquired image data;acquiring image data of an object positioned within the environment;identifying a target on the object by analyzing the acquired image data;generating images that are transmitted through from a first side of amirror to a second side of the mirror to the identified target on theobject; and projecting light onto selected surfaces in the environmentto selectively illuminate the selected surfaces, wherein a combinedimage can be formed at the identified target, the combined imagecomprising images of the selected surfaces reflected by the mirror andthe images transmitted through the mirror.
 5. The method of claim 4,wherein generating the virtual three dimensional model of the surfacesin the environment comprises determining distances to the surfaces. 6.The method of claim 4 further comprising modifying light within theenvironment to enhance the combined image.
 7. The method of claim 4,wherein identifying the target on the object is performed in real timeto modify the generated images and the projected light based onmovements of the identified target.
 8. The method of claim 4 furthercomprising determining for each of a plurality of projectors a patternof light to project to selectively illuminate the selected surfaces. 9.An apparatus comprising: a display device comprising a screen; a planarreflective surface positioned adjacent to the screen of the displaydevice, the planar reflective surface configured to allow transmissionof light from the screen through the planar reflective surface to aviewing side of the planar reflective surface; a plurality of projectorsconfigured to project light onto surfaces on the viewing side of theplanar reflective surface; an image acquisition device positioned toacquire images of the surfaces on the viewing side of the planarreflective surface; and an image blending system configured to: receiveacquired images from the image acquisition system; control the pluralityof projectors to selectively illuminate surfaces on the viewing side ofthe planar reflective surface; and control the display device toilluminate a portion of the screen and to leave a portion of the screenblank, wherein at a targeted location on the viewing side of the planarreflective surface a combined image forms, the combined image comprisingreflections of the selectively illuminated surfaces and transmittedilluminated portions of the screen.
 10. The apparatus of claim 9,wherein the plurality of projectors are adjacent to a periphery of theplanar reflective surface.
 11. The apparatus of claim 10, wherein theplurality of projectors includes four projectors positioned at cornersof the planar reflective surface.
 12. The apparatus of claim 9, whereina surface of the screen is parallel to a surface of the planarreflective surface.
 13. The apparatus of claim 9 further comprising alight configured to vary an intensity of light output based oninstructions from the image blending system.
 14. The apparatus of claim9, wherein the screen has a width that is at least 90% of the width ofthe planar reflective surface.
 15. The apparatus of claim 9, wherein thedisplay device is attached to the planar reflective surface.
 16. Theapparatus of claim 9 further comprising an active transmission matrixpositioned between the screen of the display device and the planarreflective surface.
 17. The apparatus of claim 9, wherein the screen isconfigured to output polarized light.
 18. The apparatus of claim 17,wherein the screen is configured to alternately output light oforthogonal polarizations.
 19. The apparatus of claim 9 furthercomprising a lenticular array positioned between the screen and theplanar reflective surface.
 20. The apparatus of claim 9, wherein theimage acquisition device comprises a camera configured to detectinfrared light.